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ABSTRACT 

The hierarchy of evidence based medicine postulates that systematic reviews of homogenous randomized trials 
represent one of the uppermost levels of clinical evidence. Indeed, the current overwhelming role of system- 
atic reviews, meta-analyses and meta-regression analyses in evidence based heath care calls for a thorough 
knowledge of the pros and cons of these study designs, even for the busy clinician. Despite this sore need, few 
succinct but thorough resources are available to guide users or would-be authors of systematic reviews. This 
article provides a rough guide to reading and, summarily, designing and conducting systematic reviews and 
meta-analyses. 
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"Hike to think of the meta-analytic process 

as similar to being in a helicopter. 

On the ground individual trees 

are visible with high resolution. 

This resolution diminishes as the helicopter 

rises, and in its place we begin 

to see patterns not visible from the ground" 

Ingram Olkin 

INTRODUCTION 

Systematic reviews and meta-analyses are 
being used more extensively by researchers 
and practitioners, thanks to the appeal of a 
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single piece of literature that is immediately 
able to summarize diverse data on a specific 
topic (1, 2). They have been established as 
the most quoted and read article types, even 
toppling randomized clinical trials, and 
thus they are likely to play a progressively 
even greater role in the future of medicine 
(3, 4). In addition, they are often published 
in the most prestigious international peer- 
reviewed journals, reaching thousands of 
physicians and researchers worldwide. 
As with any other analytical and research 
tool with a long-standing history {Table J), 
systematic reviews and meta-analyses, de- 
spite their major strengths, are well known 
for several potential major weaknesses. 
The aim of this review is to provide a con- 
cise but sound framework for the critical 
reading of systematic reviews and meta- 
analyses and, summarily, their design and 
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Table 1 - Key milestones in systematic review and meta-analysis development. 



Year 


XII til V 111 Lid la 


A/1 1 1 g*ctr\-ti c* 
IVAllCa LUI1C 


i on/i 


Karl rear son tUKj 


correlation between inoculation of vaccine for typhoid 
fever and mortality across apparently conflicting studies 


1931 


Leonard Tippet (UK) 


comparison of differences between and within farming 
techniques on agricultural yield adjusting for sample 
size across several studies 


1937 


William Cochran (UK) 


combination of effect sizes across different studies of 
medical treatments 


1970s 


Robert Rosenthal, Gene Glass (USA); 
Archie Cochrane (UK) 


combination of effect sizes across different studies of, 
respectively, educational/psychological and clinical tre- 
atments 


1980s 


The global scientific community 


exponential development/use of meta-analytic methods; 
birth of The Cochrane Collaboration 



conduct, stemming from our extensive ex- 
perience with this type of research method 
[Figure 1). 

Definitions 

A systematic review is a viewpoint focus- 
ing on a specific clinical problem, being it 
therapeutic, diagnostic or prognostic [Table 
2) (1, 5). The term systematic means that 
all the steps underlying the reviewing pro- 
cess are explicitly and clearly defined, and 
may be reproduced independently by other 



researchers. Thus, a formal set of meth- 
ods is applied to study search (i.e. to the 
extensive search of primary/original stud- 
ies), study selection, study appraisal, data 
abstraction and, when appropriate, data 
pooling according to statistical methods. 
Indeed, the term meta-analysis refers to a 
statistical method used to combine results 
from several different primary studies in 
order to provide more precise and valid re- 
sults. Thus, not all systematic reviews in- 
clude a meta-analysis, as not all topics are 
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Figure 1 - Publications in 
PubMed authored in the last 
few years by our research group 
concerning meta-analytic top- 
ics. PubMed was searched on 
30 March 2010 with the fol- 
lowing strategy: " (biondi-zoc- 
cai OR Zoccai) AND (meta- 
analys* OR metaanalys* 
OR metaregress* OR "meta- 
regression") ". 
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icriii 


v^iictrdLCicrisiics 


Review 


A viewpoint on a given subject quoting different primary authors or studies 


Overview 


As above 


Qualitative review 


A review which avoids a systematic approach 


Systematic review 


A review which deliberately exploits and report a systematic approach to stu- 
dy search, selection, abstraction, appraisal and pooling 


Quantitative review 


A review which deliberately exploits and report quantitative methods to eva- 
luate or synthesize data 


Meta-analysis 


A study (not necessarily a review) using specific statistical methods for poo- 
ling data from separate datasets 


Meta-regression 


A study (not necessarily a review) using specific statistical methods for explo- 
ring interactions between dependent and independent variables (moderators) 
from a meta-analysis dataset 


Individual patient data 
meta-analysis 


A study (not necessarily a review) using specific statistical methods for poo- 
ling data from separate datasets exploiting individual patient data 


Overview of reviews 


An overview of reviews which deliberately a systematic approach to review 
search, selection, abstraction, appraisal and pooling 



suitable for sound and robust data pooling. 
At the same time, meta-analysis can be con- 
ducted outside the realm of a systematic re- 
view (e.g. in the absence of extensive and 
thorough literature searches), but in such 
cases results of the meta-analytic efforts 
should be best viewed as hypothesis-gener- 
ating only. This depends mainly on the fact 
that meta-analysis outside the framework 
of a systematic review has a major risk of 
publication bias. 

Strenghts 

Systematic reviews (especially when in- 
cluding meta-analytic pooling of quantita- 
tive data) have several unique strengths 
(1, 5). Specifically, they exploit systematic 
literature searches enabling the retrieval of 
the whole body of evidence pertaining to a 
specific clinical question. 
Their standardized methods for search, 
evaluation and selection of primary stud- 
ies enable reproducibility and an objective 
stance. Individual primary studies undergo 



a thorough evaluation for internal validity, 
together with the identification of the risk 
for bias All too often, systematic reviews 
hold their greater strength precisely in their 
ability to pinpoint weaknesses and fallacies 
in apparently sound primary studies (6). 
Quantitative synthesis by means of meta- 
analysis also substantially increases statis- 
tical power, and yields narrower confidence 
intervals for statistical inference. The as- 
sessment of the effect of an intervention 
(exposure or diagnostic test) across differ- 
ent settings and times provides estimates 
and inferences with much greater exter- 
nal validity. The larger sample sizes often 
achieved by systematic reviews may even 
offer ample room for testing post-hoc hy- 
potheses or exploring the effects in selected 
subgroups (7). Clinical and statistical vari- 
ability (i.e. heterogeneity and inconsisten- 
cy) maybe exploited by advanced statistical 
methods such as meta-regression, possibly 
offering the opportunity to test novel and 
hitherto unprecedented hypotheses (8). Fi- 
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164 nally, meta-regression methods can be used 
to perform adjusted indirect comparisons 
or network meta-analyses (9). 

Limitations 

Drawbacks of systematic reviews and me- 
ta-analyses are also substantial, and should 
never be dismissed (1). Since the first cri- 
tique of being "an exercise in mega-silliness" 
and inappropriately "mixing apples and or- 
anges" (10), there has been ongoing debate 
on the most correct approach to choose 
when meta-analytic pooling should be pur- 
sued (e.g. in case of statistical homogeneity 
and consistency) and when, conversely, the 
reviewer should refrain from meta-analysis 
(e.g. in case of severe statistical heterogene- 
ity [as testified by p values < 0.10 at x 2 test] 
or significant statistical inconsistency [as 
testified by I 2 values > 50 % ]) (11). 
Whereas Canadian authors suggest that 
systematic reviews and meta-analyses 
from homogenous randomized controlled 
trials represent the apex of the evidence- 
based medicine pyramid (discounting for 
the role of n of 1 randomized trials) (12), 
others maintain that very large and sim- 



ple randomized clinical trials offer several 
premium features, and should always be 
preferred, when available, to systematic re- 
views (13). 

It is also all too common to retrieve only a 
few studies which focus on a given clinical 
topic, or otherwise studies may be found, 
but of such low quality, that including or 
even discussing them in the setting of a 
systematic review may appear misleading. 
Indeed, in such cases the meta-analysis it- 
self can be considered misleading. None- 
theless, key insights may be gained in these 
cases by exploring sources of heterogeneity, 
stratified analyses, and meta-regressions. 
This drawback is strictly associated with 
the major threat to meta-analysis validity 
called the small study effect (also, albeit 
inappropriately, called small study bias or 
publication bias) (1). Indeed, it is common 
to recognize, especially in large datasets, 
that small primary studies are more likely 
to be reported, published and quoted if their 
results are significant. Conversely, small 
non-significant studies often fail to reach 
publication or dissemination, and may thus 
be very easily missed, even after thorough 



Greater flexibility - Lower validity 



Qualitative reviews 



Systematic reviews 



Meta-analyses from 
individual studies 



Meta-analyses from 
individual patient data 



Case reports and series 



Observational studies 



Observational 
controlled studies 



Randomized controlled trials 



Multicenter randomized 
controlled trials 



Lower flexibility - Greater validity 



Figure 2 - Parallel hierarchy 
of scientific studies in clinical 
research. Modified from Bion- 
di-Zoccai et al(2). 
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literature searches. Combining results from 
these "biased" small studies with those of 
larger studies (which are usually published 
even if negative or non-significant) may 
inappropriately deviate summary effect 
estimates away from the true value. Unfor- 
tunately, despite the availability of several 
graphical and analytical tests (14) small 
study effects (which actually encompass 
publication bias) are potentially always 
present in a systematic review and should 
never be disregarded. 

In addition, in the ongoing worldwide re- 
search effort, it is all too common for re- 
viewers to focus only on English language 
studies, and thus unduly restricting their 
search and excluding potentially important 
works (e.g. from China or Japan). Another 
common critique is that systematic reviews 
and meta-analyses are not original research 
[Figure 2). The reader is left free to form in- 
dependently his informed opinion on this 
specific issue. Nonetheless, the main meter 
to judge a systematic review should be its 
novelty and usefulness for the very same 
reader, not whether it appears as original 
or secondary research (2). 
Finally, a burning issue is whether results 
from large systematic reviews and meta- 
analyses can ever be applied to the single 
individual under our care. This question 
cannot be answered once and for all, and 
judgment should always be employed when 
considering the application of meta-ana- 
lytic results to a specific patient. Unless 
proven otherwise by a significant test of in- 
teraction, all patients should be considered 
likely to similarly benefit from a specific 
treatment or diagnostic strategy (12). 

Appraising primary studies, systematic 
reviews and meta-analyses 

Unfortunately publication of a systematic 
review in a peer-reviewed journal is not 
definitive evidence of its internal validity 
and usefulness for the clinical practitioner 



or researcher (15). Peer-review is not very 
accomplished in judging or improving the 
quality of scientific reviews, and many ex- 
amples of bad or unsuccessful peer-review- 
ing efforts can be easily found. However, 
just as "democracy is the worst form of gov- 
ernment except all those other forms that 
have been tried" (Sir Winston Churchill), 
peer-review is the "worst" method used to 
evaluate scientific research except all other 
methods that have been tried so far. This 
applies to all clinical research products in 
general and so also applies to systematic 
reviews and meta-analyses. Thus, provided 
that meta-analyses are accurately and thor- 
oughly reported, the burden of quality ap- 
praisal lies largely, as usual, in the eye of 
the beholder (i.e. the reader). 
Assessment of primary research studies as 
well as systematic reviews and meta-analy- 
ses should be based on their internal valid- 
ity and then, provided it is reasonably ade- 
quate, on their results and external validity 
(12). Whereas interpretation of results and 
external validity of any research endeavor 
depends on the specific context of applica- 
tion, and is thus best left open to the indi- 
vidual judgment of the reader or decision- 
maker, internal validity can be evaluated in 
a rather structured and validated way. Re- 
cent guidance on the appraisal of the risk 
of bias in primary research studies within 
the context of a systematic review has been 
provided by The Cochrane Collaboration, 
and includes a separate assessment of the 
risk of selection, performance, attrition 
and adjudication bias [Table 3) (16). Other 
valid and complementary approaches, tar- 
geted for specific study designs, have been 
proposed by advocates of evidence-based 
medicine methods, and include the Jadad 
score, the Delphi list, and the Megens- 
Harris list (12). Nonetheless, even external 
validity can be formally evaluated by focus- 
ing on the population included, the control 
group, and result interpretation. Finally, 
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Table 3 - A modified version of The Cochrane Collaboration risk of bias assessment tool for the appraisal of primary 



studies. (1 6) * 



Question 


Answers 


Meaning 


Adequate sequence 
generation? 


Yes, no, or uncertain 


Was the allocation sequence generated appropria- 
tely (eg computer or table of random numbers)? 


ililUuClLlUll v y 1 1 v> w CI 1 1 1 1 ^ 1 1 L U.OvU. a 


Vpq tin nr nnrprtain 


Were physicians unaware of allocation code up to 
actual patient enrolment? 


Blinding? 


Yes, no, or uncertain 


Were patients, caregivers, outcome assessors, an- 
cillary personnel, and/or statisticians unaware of 
actual treatment? 


Concurrent therapies similar? 


Yes, no, or uncertain 


Were concurrent medical and non-medical treat- 
ments similar in the groups under comparison? 


Incomplete outcome data 
addressed? 


Yes, no, or uncertain 


Were all data analyzed, minimizing the impact of 
losses to follow-up? 


Uniform and explicit outcome 
definitions? 


Yes, no, or uncertain 


Were definitions clearly spelled out and employed 
consistently to adjudicate events or outcomes? 


Free of selective outcome 
reporting? 


Yes, no, or uncertain 


Were all relevant outcomes thoroughly reported? 


Free of other bias? 


Yes, no, or uncertain 


Was the risk of any other bias low? 


Overall risk of bias? 


High, moderate, 
or low 


What is the comprehensive assessment of the risk 
of bias of the study? 



*The following items are not present in the original version supported by The Cochrane Collaboration: concurrent therapies 
similar; uniform and explicit outcome definitions; overall risk of bias 



Table 4 - Oxman and Guyatt index for the appraisal of reviews. (19) * 



Question 


Details 


1 


Were the search methods used to find evidence stated? 


2 


Was the search for evidence reasonably comprehensive? 


3 


Were the criteria for deciding which studies to include in the overview reported? 


4 


Was bias in the selection of studies avoided? 


5 


Were the criteria used for assessing the validity of the included studies reported? 


6 


Was the validity of all studies referred to in the text assessed using appropriate criteria 


7 


Were the methods used to combine the findings of the relevant studies reported 


8 


Were the findings of the relevant studies combined appropriately relative to the primary 
question the overview addresses? 


9 


Were the conclusions made by the author (s) supported by the data and/or analysis re- 
ported in the overview? 


10 


This question summarizes the previous ones and, specifically, asks to rate the scientific 
quality of the review from 1 (being extensively flawed) to 3 (carrying major flaws) to 5 
(carrying minor flaws) to 7 (minimally flawed) 



*The Oxman and Guyatt index evaluates the internal validity of a review on 9 separate questions for which 3 distinct an- 
swers are eligible ("yes", "partially/can't tell", "no"). The developers of the index specify that if the "partially/can't tell" 
answer is used one or more times in questions 2, 4, 6, or 8, a review is likely to have minor flaws at best and is difficult to 
rule out major flaws (ie a score<4). If the "no" option is used on question 2, 4, 6 or 8, the review is likely to have major flaws 
(ie a score<3) 
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established statistical criteria are available 
to determine whether a given intervention 
is effective and similar explicit criteria can 
inform on the presence of clinical signifi- 
cance. 

The quality of a systematic review and me- 
ta-analysis depends on several factors, in 
particular the quality of the primary pooled 
studies. 

Nonetheless, reporting quality (e.g. com- 
pliance with current guidelines on draft- 
ing and reporting of a meta-analysis by the 
Preferred Reporting Items for Systematic 
reviews and Meta-Analyses [PRISMA] or 
Meta-analysis Of Observational Studies 
in Epidemiology [MOOSE] statements) 
should be clearly distinguished by internal 
validity (17, 18). This can be low even in 
well reported reviews, whereas it is gen- 
erally difficult to judge as highly valid a 
poorly reported systematic review and 
meta-analysis. The assessment of the in- 
ternal validity of a review is quite complex 
and based on several assumptions, includ- 
ing study search and appraisal, methods for 
data pooling, and approaches to interpreta- 
tion of study findings. 



However, useful guidance was provided by 
Oxman and Guyatt with their well validat- 
ed instrument (Table 4) (19). 
More recently, other investigators have 
suggested other tools for the evaluation of 
systematic reviews, such as the A Measure- 
ment Tool to Assess Systematic Reviews 
(AMSTAR), and the Veritas plot, which 
await further validation (Table 5, Figure 3) 
(20-22). 

For those busy critical care physicians 
wishing for a quicker approach to appraise 
systematic reviews, a simple two-step ap- 
proach can be proposed. This is a simpli- 
fication of the evidence-based medicine 
approach for the evaluation of sources of 
clinical evidence, but is nonetheless quite 
helpful (12). Evidence-based medicine is 
"the conscientious, explicit, and judicious 
use of current best evidence in making deci- 
sions about the care of individual patients" 
(12). It must also be stressed that "the prac- 
tice of evidence-based medicine requires 
integration of individual clinical expertise 
and patient references with the best avail- 
able external clinical evidence from sys- 
tematic search" (12). Systematic reviews 
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Table 5 - The AMSTAR tool for the appraisal of systematic reviews. (20-21) * 



Question 


Details 


1 


Was an 'a priori' design provided? 


2 


Was there duplicate study selection and data extraction? 


3 


Was a comprehensive literature search performed? 


4 


Was the status of publication (i.e. grey literature) used as an inclusion criterion? 


5 


Was a list of studies (included and excluded) provided? 


6 


Were the characteristics of the included studies provided? 


7 


Was the scientific quality of the included studies assessed and documented? 


8 


Was the scientific quality of the included studies used appropriately in formula- 
ting conclusions? 


9 


Were the methods used to combine the findings of studies appropriate? 


10 


Was the likelihood of publication bias assessed? 


11 


Was the conflict of interest stated? 



*The AMSTAR (a measurement tool to assess the methodological quality of systematic reviews), evaluates the quality of 
a review on 11 separate questions for which 4 distinct answers are eligible ("yes", "no", "can't answer", "not applicable") 
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and meta-analyses, if well conducted and atic review and meta-analysis is to try and 



reported, help us in reducing our efforts in find an answer to the question: can I trust 

looking for, evaluating, and summarizing it? In other words, is this review internally 

the evidence. valid, does it provide a precise and largely 

But the burden of deciding what to do with unbiased answer to its scientific question? 

the evidence obtained for the care of our in- Providing a definitive assessment of the 

dividual patient remains ours. internal validity of a systematic review is 

Thus, the first step in appraising a system- not a simple task, but largely depends on 
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Figure 3 - Typical diagram used to generate a Veritas plot (panel A) (22). 

Using this tool, a low quality meta-analysis will be represented by a hexagon with a smaller area (panel 
B), whereas a high quality meta-analysis will be shown as a hexagon with a larger area (panel C). 
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the methods employed and reported regard- 
ing study search, selection, abstraction, ap- 
praisal and, if appropriate, the study pool- 
ing. Even if we can conclude that a given 
meta-analysis is internally valid, we still 
have to face the second step in its evalua- 
tion. This focuses on the external validity 
of the study. In other words, can I apply 
the review results to the case I am facing 
or will shortly face? More basically it an- 
swers the question: so what? Decisions on 
external validity are highly subjective and 
may change depending on the clinical, his- 
torical, logistical, cultural or ethical context 
of the evaluator. Nonetheless, systematic 
reviews and meta-analyses can improve 
our appraisal of the external validity of any 
given clinical intervention, by suggesting 
an overall clinical efficacy (or lack of it). 
It is clear that the assessment of the internal 
validity, and even more importantly the ex- 
ternal validity, of any research endeavor, is 
highly subjective, and thus we leave ample 
room for the reader to enjoy and appraise 
them on his or her own. 
The only issue that is worth being fur- 
ther stressed is that only collective and 
constructive, but critical post-publication 
evaluation of scientific studies can put and 
maintain them into the appropriate context 
for their correct and practical exploitation 



by the clinical researcher and the clinical 
practitioner. 

Systematic reviews and meta-analyses: 
do it yourself 

Even those not strictly committed to con- 
duct a systematic review may obtain further 
insights into this clinical research method 
by understanding the key steps involved in 
the design, conduct and interpretation of a 
systematic review (5). 
Briefly, a systematic review should always 
stem from a specific clinical question. 
Even if the experienced reviewer can prob- 
ably informally guess the answer to this 
question the goal of the systematic review 
will be to confirm or disprove such hypoth- 
esis in a formal and structured way. With 
this goal in mind, the review should be 
designed as prospectively and in as much 
in detail as possible, to avoid conscious or 
unconscious manipulations of methods or 
data {Figure 4) . 

The next steps are very important, and 
define the boundaries of the reviewing ef- 
fort. Specifically, the reviewer should spell 
out the population of interest, the inter- 
vention or exposure to be appraised, the 
comparison(s) or comparator(s), and the 
outcome (s). The acronym PICO is often 
used to remember this approach. As an 
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Definition of question and hypothetical solution +- 

Prospective design of the systematic review 

Problem formulation (population, intervention 
or exposure, comparison, outcome [PICO]) 

Data search 

Data abstraction and appraisal 

Data analysis + quantitative synthesis 

Result interpretation and dissemination 



Figure 4 - Typical algorithm 
for the design and conduct of 
a sgstematic review. Modified 
from Biondi-Zoccai et at (5). 
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170 example, we could be interested in con- 
ducting a systematic review focusing on a 
population (P) of diabetics with coronary 
artery disease undergoing coronary artery 
bypass grafting, with the intervention (I) 
of interest being the administration of bi- 
varidudin as anticoagulant, the comparator 
(C) being unfractioned heparin, and the 
outcomes (O) defined as in-hospital rates 
of death, myocardial infarction, stroke, or 
major bleeding (including bleeding needing 
repeat surgery). 

After such preliminary steps, the actual re- 
view begins with a thorough and extensive 
search, encompassing several databases 
(not only MEDLINE/PubMed) with the 
help of library personnel experienced in 
literature searches, preferably also includ- 
ing conference abstracts and bibliographies 
of pertinent articles and reviews. When a 
list of potentially pertinent citations has 
been retrieved, these should be assessed 
and included/excluded based on criteria 



stemming directly from the PICO approach 
used to define the clinical question. Study 
appraisal also includes a formal evaluation 
of study validity and risk of bias of primary 
studies, whereas data abstraction, gener- 
ally performed by at least two independent 
reviewers with divergences resolved after 
consensus, provides the quantitative data 
which will eventually be pooled with meta- 
analysis (16). 

Indeed, provided that studies are relatively 
homogeneous and consistent, meta-analyt- 
ic methods are employed to combine effect 
estimates from single studies into a unique 
summary effect estimate, with correspond- 
ing p values and confidence intervals for the 
effect {Figure 5). In many cases results may 
lead reviewers to go back to the original re- 
search question and revise their working 
hypothesis. The last step relies on the in- 
terpretation and dissemination (possibly 
through publication in a peer-reviewed 
journal) of the results. 



Review: Fenoldopam for reno-protection 
Comparison: Fenoldopam vs control Rx 
Outcome: Death 



Study 

or sub-category 



Biancofiore 
Delia Rocca 
Morelli I 
Morelli II 
Bove 
Tumlin 

Total (95% CI) 




Control 

n/N 



357 



2/94 
3/21 
9/19 
66/150 
4/40 
19/75 

399 



Total events: 79 (Fenoldopam), 103 (Control) 
Test for heterogeneity: Chi 2 = 1.51, df = 5 (P = 0.91), I 2 = 0 % 
Test for overall effect: Z = 2.19 (P = 0.03) , 



OR (fixed) 
95% CI 



OR (fixed) 
95% CI 



1.02 [0.09, 11.57] 
0.60 [0.09; 4.01] 
1.00 [0.28, 3.57] 
0.68 [0.42, 1.08] 
1.00 [0.23, 4.31] 
0.47 [0.21, 1.07] 

0.67 [0.47, 0.96] 



0.2 0.5 



Figure 5 - Typical forest plot generated bg RevMan from a sgstematic review with meta-analgtic pool- 
ing of dichotomous outcomes (df= degrees of freedom; E = expected cases; O = observed cases; OR = odds 
ratio). The solid oval highlights event counts in one of the groups under comparison, the solid box shows 
graphicallg individual and pooled point effect estimates with 95 % confidence intervals, the arrowhead 
indicates the exact pooled point effect estimate with 95 % confidence intervals (CI), the arrow shows the 
p value for effect, and the dashed oval highlights p value for statistical heterogeneitg and measure of sta- 
tistical inconsistencg (12). Modified from Landoni et al. (30). 
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More advanced analytical issues 

Unless extensively powered low event rates 
may often be found in primary research 
studies (e.g. with > 1000 patients enrolled 
or with selective recruitment of very high- 
risk subjects). This may lead to null counts 
in one or more of the groups undergoing 
comparison in a controlled trial, generating 
severe computational hurdles. Indeed, most 
statistical methods used for meta-analytic 
pooling require that at least one event has 
occurred in each study group. 
When this is not the case in one or more 
of the groups under comparison, bias may 
be introduced with the common practice of 
adding 0.25 or 0.50 to each group without 
events (23). On top of this, when no event 
has occurred in any group, comparisons 
are more challenging and data from such 



an underpowered studies cannot be pooled 
with standard meta-analytic methods, as 
variance of the effect estimate approaches 
infinity. Nonetheless, other approaches 
(e.g. risk difference, continuity correction 
or Peto method) can still be used in case of 
total zero event trials. 
Even when all groups undergoing compar- 
ison in a specific study have one or more 
events, the risk of biased estimates and al- 
pha error (i.e. the risk of erroneously dis- 
missing a null hypothesis despite it being 
true) maybe present (1). 
Indeed, minor differences in populations 
with few and rare events may provide nom- 
inally significant results (e.g. p = 0.048) 
which however appear quite unstable. In 
such cases, we recommend reliance on the 
combined use of p values and 95% confi- 
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Figure 6 - Typical funnel plot generated by RevMan showing small study bias, ie the asymmetric distri- 
bution of effect sizes in function of study precision, with selective publication of only positive small sample 
studies (RR = relative risk; SE = standard error). Modified from Biondi-Zoccai et at (24). 
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172 dence intervals, or even making use of 99 % 
confidence intervals. In other cases, a use- 
ful rule of thumb is to trust only meta-anal- 
yses reporting on at least 100 pooled events 
per group under comparison. 
The risk of erroneously accepting a null hy- 
pothesis despite it being false (i.e. the beta 
error) is also common in systematic re- 
views and meta-analyses, especially when 
they include few studies with low event 
counts. This lack of statistical power (de- 
fined as 1-beta) is even more common with 
meta-regression analyses, which are usu- 
ally underpowered because of few included 
studies and regression to the mean (7). 
Surrogates may provide an important con- 
tribution to clinical research design, by in- 
creasing statistical power and offering in- 
sights in more than one clinical dimension. 
However, surrogate end-points (e.g. > 25 % 
increase in serum creatinine from baseline 
values to identify subclinical renal injury) 
may be less clinically relevant than hard 
clinical end-points (death or permanent 
need for hemodialysis) (12). 
Usually, only surrogates which have a di- 
rect impact on patient well being and are 
independently associated with hard clini- 
cal end-points should be accepted for the 
design of clinical research studies. In any 
case, a study reaching significance based 
on surrogate end-points alone, but missing 
significance on analysis of hard end-points 
should be considered as hypothesis-gener- 
ating or, at best, underpowered. 
Small study bias always potentially threat- 
ens the results of a systematic review, as 
this type of confounding applies to all clini- 
cal topics and research study designs {Fig- 
ure 6) (24). 

Although this bias may be less significant 
in more recent and well financed drug or 
device studies (e.g. fenoldopam), in older 
or less well funded studies publication bias 
may profoundly undermine the results of a 
systematic review. 



This has been all too evident in studies ex- 
amining the role of acetylcysteine for the 
prevention of contrast-associated nephropa- 
thy (25), but is also obvious in other com- 
monly prescribed agents. Another major 
threat to the validity of a systematic review, 
as to any other research endeavor, lies in 
conflicts of interest and study funding. It is 
well known that reviewers with underlying 
financial conflicts of interest are more likely 
to conclude in favor of the intervention ben- 
efiting the source of financial gains (26). 
Whether these facts should lead to a more 
critical reading of their work or a com- 
prehensive re-evaluation of their whole 
research project is best left at the readers' 
discretion, but this should also take into 
account the overall internal validity (e.g. 
blinding of patients, physicians, adjudica- 
tors, and analysts) of the work. 



CONCLUSIONS 

Systematic reviews and meta-analyses offer 
powerful methods to evaluate the clinical 
effects of health interventions, especially 
when directly applied to real world clini- 
cal practice (such as in the Best Evidence 
Topic [BET] approach) (27). 
More collaborative efforts are however re- 
quired to design, conduct and disseminate 
individual patient data meta-analyses in an 
unbiased and rigorous manner (28, 29). 
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